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1 SYSTEM AND METHOD FOR OPERATOR ASSISTED AU TOMATED CAT T 

2 HANDLING 

3 

4 BACKGROUND OF THE INVENTION 

5 1. Field of the Invention 

6 The present invention relates generally to systems apparatus and methods for 

7 automated call handling, and more particularly to a system and method for operator 

8 assisted automated call handling. 

9 2. Discussion of Background Art 

10 Automated call handling systems, such as Interactive Voice Response (IVR) 

1 1 systems, using Automatic Speech Recognition (ASR) and Text-to-speech (TTS) 

12 software are increasingly important tools for providing information and services in a 

13 more cost efficient manner. IVR systems are typically hosted by a server that includes 

14 an array of Digital Signal Processors (DSPs), and enable users to interact with 

15 corporate databases and services over a telephone using a combination of voice 

16 utterances and telephone button presses. IVR systems are particularly cost effective 

17 when a large number of users require data or services that are very similar in nature 

18 and thus can be handled in an automated manner often providing a substantial cost 

19 savings due to a need for fewer human operators. 

20 In an ideal situation, an IVR system would be able to automatically guide a 

21 user through an entire transaction using only predefined dialogs, without any human 

22 interference. In reality, however, since speech recognition technology is still not 

23 perfect and perhaps will not be perfect for decades to come, from time to time, the 

24 user has to resort to a human operator after numerous unsuccessful communication 

25 efforts with the IVR system. 
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1 For example, while most users can put up with the machine-like TTS 

2 utterances and even the rigid dialog mode (i.e. step by step with a lot of 

3 confirmations), most users will not tolerate an IVR system that cannot recognize their 

4 voice response, even though they have repeatedly tried their best to speak it many 

5 times and as clearly as they can. Although the computer can be very patient and polite 

6 saying, "I'm sorry, but I cannot understand what you are saying. Would you please 

7 repeat?" users will get frustrated and form negative impressions of IVR systems, even 

8 if they are served well by a human operator in the end. Such users encountering IVR 

9 systems in the future often think that these systems are just too primitive, and thus will 

10 directly seek human assistance from the beginning of their call, and avoid the IVR 

11 system. 

12 Toward this end, some IVR systems provide a hot key (for example, "#" key) 

13 or voice command (for example, "Help") so that users can be connected to a human 

14 operator when there are problems. Once a user is transferred to a human operator the 

15 IVR system is terminated with respect to that user, and the human operator completes 

16 the transaction with the user. If such transfers from the IVR system to a human 

17 operator occur too often, the benefits of having an IVR system are reduced. 

18 In response to the concerns discussed above, what is needed is an automated 

19 call handling system and method that overcomes the problems of the prior art. 
20 
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1 SUMMARY OF TRF. INVENTION 

2 The present invention is a system and method for operator assisted automated 

3 call handling. The method of the present invention includes the elements of: initiating 

4 a dialog between a user and an automated call handling system; receiving input from a 

5 user in response to an input request by the system; sending a human operator the user 

6 input, if the system can not interpret the user input; providing the system with an 

7 interpreted response generated by the operator based on the user input; continuing the 

8 dialog between the user and the system if the interpreted response is confirmed by the 

9 user; and connecting the user to the operator if the interpreted response is not 

10 confirmed by the user. The system of the present invention includes all means for 

11 practicing the method. 

12 These and other aspects of the invention will be recognized by those skilled in 

13 the art upon review of the detailed description, drawings, and claims set forth below. 
14 
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1 BRIEF DESC RIPTION OF THE PR AWfljGS 

2 Figure 1 is a dataflow diagram of one embodiment of a system for operator 

3 assisted automated call handling; and 

4 Figures 2A through 2C are a flowchart of one embodiment of a method for 

5 operator assisted automated call handling. 
6 
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1 DFTATT ED DESCRIPTION OF THE PREFERRED EMBODIMENT 

2 The present invention addresses the problem where automated call handling 

3 systems are unable to resolve a user's speech signal utterance. In such instances, the 

4 present invention routes the speech signal to a human operator who then resolves the 

5 speech signal into a user response and provides that response back to the call handling 

6 system. Preferably human operators are used only to resolve a user's hard-to- 

7 understand utterances that Automatic Speech Recognition (ASR) software in the call 

8 handling system can not. The user need not be aware that the operator has helped or 

9 was involved at all and preferably continues to use the call handling system after the 

10 operator's help. This should improve a user's satisfaction with and continued use of 

11 such automated systems. The present invention also lowers call center costs, since the 

12 human operators are used mainly for specific portions of a user's call, and need not 

13 take over the entire call each time the call handling system can not resolve a user's 

14 utterance. The present invention preferably operates in conjunction with an 

15 Interactive Voice Response (IVR) system. 
16 

17 Figure 1 is a dataflow diagram of one embodiment of a system 100 for 

18 operator assisted automated call handling. In response to a user 102 contacting the 

19 call handling system 104, a dialog record 106 specific to the user 102 is created. A 

20 dialog manager 108 routes user and system 104 information through a dialog router 

21 1 10 to the dialog record 106. The dialog record 106 stores a record of interactions 

22 between the user 102 and the call handling system 104. These interactions are also 

23 preferably labeled as a dialog between the user 102 and the system 104, but 

24 alternatively may be labeled as a session or a transaction. During normal operation of 

25 the present invention, multiple instances of the dialog record 106 are likely as many 

26 users contact the call handling system 104 simultaneously. 

5 
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1 The call handling system 104 creates and maintains a dialog state 1 12 and a 

2 dialog data cache 1 14 within the dialog record 106. The dialog state 112 includes a 

3 set of pointers that indicate a current state and a set of historic user states within the 

4 call handling system 104. The set of pointers also includes a set of path pointers that 

5 indicate how the user 102 reached the current user state within the call handling 

6 system 104. For example, the user 102 may have already gone through one branch of 

7 a question and answer tree that requested that the user 102 identify themselves to the 

8 call handling system 104. The dialog state 1 12 will have a record of this branch. The 

9 user 102 may now be at a current call handling system state where the user 102 is 

10 permitted to request a set of services hosted by or data from the call handling system 

11 104. 

12 The dialog data cache 1 14 can include data input by the user 102, such as a set 

13 of user answers in response to call handling session 104 questions, a set of telephone 

14 key presses, and or a set of user speech signal utterances. The dialog data cache 1 14 

15 also can include data output to the user 102, such as bank records, outstanding 

16 customer service items, as well as many other types of data that can be processed by 

17 the call handling system 104. The call handling system 104 draws on Automatic 

18 Speech Recognition (ASR) and Text-to-speech (TTS) software modules (not shown) 

19 at various times in order to interpret the user's speech signals, maintain the dialog 

20 state 1 12, store information in the dialog data cache 1 14, and generate audible call 

21 handling system 104 responses that the user 102 can listen to. 

22 The dialog manager 108 creates a user specific instance of an Item Wide 

23 Frustration Index 1 16 and a predetermined Item Wide Frustration Threshold 118, 

24 within the dialog record 106. The Item Wide Frustration Index 1 16 is a number that is 

25 incremented and reset by the dialog manager 108 in response to a predetermined set of 

26 events occurring within the call handling system 104. The Item Wide Frustration 
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1 Index 1 16 is intended to correspond to an instantaneous user frustration level at a 

2 current state of the dialog record 106. The dialog manager 108 resets the Item Wide 

3 Frustration Index 1 16 to an initial value, such as zero, upon first contact with the call 

4 handling system 104 by the user 102. 

5 The Item Wide Frustration Threshold 1 18 can be selected in many different 

6 ways. In a first embodiment, the dialog manager 108 can set the Item Wide 

7 Frustration Threshold 1 18 to a fixed value, such as three. In a second embodiment, 

8 the dialog manager 108 can look up an importance attribute associated with the user 

9 102 and set the Item Wide Frustration Threshold 1 18 to a predetermined low value, if 

10 the user's 102 importance attribute is of a predetermined high value. The dialog 

11 manager 108 can set the Item Wide Frustration Threshold 1 18 to a predetermined high 

12 value, if the user's 102 importance attribute is of a predetermined low value. 

13 In a third embodiment, the dialog manager 108 can look up a personality 

14 attribute associated with the user 102 and set the Item Wide Frustration Threshold 1 18 

15 to a predetermined low value, if the user's 102 personality attribute is of a 

16 predetermined low frustration tolerance value. The dialog manager 108 can set the 

17 Item Wide Frustration Threshold 1 18 to a predetermined high value, if the user's 102 

18 personality attribute is of a predetermined high frustration tolerance value. The 

19 frustration tolerance level can also be interpreted as a user patience level. 

20 In a fourth embodiment, the dialog manager 108 can look up a physical 

21 attribute associated with the user 102 and set the Item Wide Frustration Threshold 1 18 

22 to a predetermined low value, if the user's 102 physical attribute is of a first 

23 predetermine value. The dialog manager 108 can set the Item Wide Frustration 

24 Threshold 1 18 to a predetermined high value, if the user's 102 physical attribute is of 

25 a second predetermined value. Physical attributes can include a user's age, gender, 

26 and so on. 

7 
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1 In a fifth embodiment, the dialog manager 108 can look up a call connection 

2 attribute associated with the user 102 and set the Item Wide Frustration Threshold 118 

3 to a predetermined low value, if the user's 102 call connection attribute is of a first 

4 predetermine value. The dialog manager 108 can set the Item Wide Frustration 

5 Threshold 1 18 to a predetermined high value, if the user's 102 call connection 

6 attribute is of a second predetermined value. Call connection attributes can include 

7 whether or not the user 102 is calling from a mobile phone, whether or not the user is 

8 driving in a car, and so on. Call connection attributes can be determined in some 

9 cases by look-up tables by user and in other cases by using signal pre-analysis. 

10 And, in a sixth embodiment, the dialog manager 108 can look up an operator 

11 availability attribute associated with the call handling system 104 and set the Item 

12 Wide Frustration Threshold 1 18 to a predetermined low value, if a relatively lower 

13 number of human operators are available to help the user 102. The dialog manager 

14 108 can set the Item Wide Frustration Threshold 118 to a predetermined high value, if 

15 a relatively higher number of human operators are available to help the user 102. 

16 Usually, if there are a large number of operators available to assist the user 102 the 

17 threshold 118 will be set lower than if there were only a small number of available 

18 operators. 
19 

20 From time to time the call handling system 104 will request that the user 102 

21 respond to the system 104 in some way. The requested response may solicit from the 

22 user 102 a button press, a vocal utterance, or other form of user response. The user 

23 102 transmits the user response to the call handling system 104. If the call handling 

24 system 104 can not interpret the user response, the dialog manager 108 increments the 

25 Item Wide Frustration Index 116, and compares the Item Wide Frustration Index 1 16 

26 with the predetermined Item Wide Frustration Threshold 1 18. If the Item Wide 
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Frustration Index 1 16 is below the predetermined Item Wide Frustration Threshold 
118, the dialog manager 108 instructs the call handling system 104 to repeat the user 
response request. 

If the Item Wide Frustration Index 1 16 is at or above the predetermined Item 
Wide Frustration Threshold 118, the dialog manager 108 does not interrupt the dialog 
between the user 102 and the system 104, but instead: first, sends a human operator 
120 the current state information stored in the dialog state 1 12; second sends the 
operator 120 a relevant set of user responses, such as the user's speech signal, stored 
in the dialog data cache 1 14; and third resets the Item Wide Frustration Index 116 
back to it's initial value. The current state information provides the operator 120 with 
contextual information with regard to the current state of the dialog between the user 
102 and the call handling system 104. The operator 120 then examines the current 
state information and the user responses. For instance, the contextual information 
may include a question requiring the user 102 to enter vacation destination name. 
Next, the operator 120 listens to the cached user responses. 

If the operator 120 can interpret the user responses, the operator 120 provides 
the call handling system 104 with an interpreted response. The call handling system 
104 asks the user 102 if the interpreted response corresponds to the user's intended 
response. If the interpreted response corresponds to the user's intended response, the 
call handling system 104 continues interacting with the user 102. The user 102 need 
not even know that the operator 120 was involved in generating the interpreted 
response. 

If the interpreted response does not correspond to the user's intended response, 
the dialog manager 108 interrupts the dialog between the user 102 and the system 104 
and commands the dialog router 1 10 to connect the user 102 directly to the operator 
120. The operator 120 then enters in to a normal human dialog with the user 102 in 
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1 order to determine what the user's 102 intended response was. Based on the 

2 operator's 120 judgment, the user 102 may or may not be passed back to the call 

3 handling system 104 in order to complete the user's transaction. 

4 The dialog manager 108 also commands the dialog router 1 10 to connect the 

5 user 102 directly to the operator 120 at any point in the dialog, if the user 102 begins 

6 to speak out of context. An example of out of context user information includes when 

7 the user 102 starts to complain about the foolishness of the call handling system 104 

8 after getting impatient. Out of context information can be detected by comparing the 

9 user's 102 responses to a known set of predefined words associated with user 
10 frustration. 

11 

12 The call handling system 104 creates a user specific instance of a Transaction 

13 Wide Frustration Index 122, and a set of Transaction Wide Frustration Thresholds 

14 124, within the dialog record 106. The Transaction wide Frustration Index 120 is a 

15 number that is incremented and reset by the dialog manager 108 in response to a 

16 predetermined set of events occurring within the call handling system 104. The 

17 Transaction Wide Frustration Index 122 is intended to correspond to an overall user 

18 frustration level with the call handling system 104, as recorded within the entire 

19 dialog record 106. The dialog manager 108 resets the Transaction Wide Frustration 

20 Index 122 to an initial value, such as zero, upon first contact with the call handling 

21 system 104 by the user 102. 

22 The Transaction Wide Frustration Thresholds 124 can be selected in many 

23 different ways, including those discussed above with respect to the Item Wide 

24 Frustration threshold. If the Item Wide Frustration threshold 1 18 has been reached or 

25 exceeded, the dialog manager 108 increments the Transaction Wide Frustration Index 
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1 122, and compares the Transaction Wide Frustration Index 122 with a first 

2 Transaction Wide Frustration Threshold within the set of thresholds 124. 

3 If the Transaction Wide Frustration Index 122 is below the first Transaction 

4 Wide Frustration Threshold, the dialog manager 108 does not interrupt the call 

5 handling system's 104 dialog with the user 102. If the Transaction Wide Frustration 

6 Index 122 is at or above the first Transaction Wide Frustration Threshold, the dialog 

7 manager 108 transmits a warning signal to the operator 120 and provides the operator 

8 120 with an option to interrupt the dialog and directly connect to the user 102 through 

9 the dialog router 1 10. If the Transaction Wide Frustration Index 122 is at or above a 

10 second Transaction Wide Frustration Threshold, which is higher than the first 

11 Transaction Wide Frustration Threshold, the dialog manager 108 automatically 

12 commands the dialog router 1 10 to connect the user 102 directly to the operator 120. 

13 The operator 120 then enters in to a normal human dialog with the user 102. 

14 Data collected by the present invention may also be used as a ground-truthing 

15 engine for improving the call handling system 104 and setting the thresholds. 
16 

17 Figures 2 A through 2C are a flowchart of one embodiment of a method 200 

18 for operator assisted automated call handling. The method 200 begins in step 202, 

19 where a user 102 contacts and enters into a dialog with the call handling system 104. 

20 In step 204, the call handling system 104 creates an instance of a dialog record 106 for 

21 that user 102, and in step 206, a dialog manager 108 routes the dialog information to 

22 the dialog record 106 through a dialog router 1 10. 

23 I" step 208, the call handling system 104 creates and maintains a dialog state 

24 1 12 and a dialog data cache 1 14 within the dialog record 106. In step 210, the dialog 

25 manager 108 creates a user specific instance of an Item Wide Frustration Index 116 
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1 and a predetermined Item Wide Frustration Threshold 118, within the dialog record 

2 106. 

3 In step 212, the dialog manager 108 resets the Item Wide Frustration Index 

4 1 16 to an initial value, such as zero, upon first contact with the call handling system 

5 104 by the user 102. In step 214, the dialog manager 108 sets the Item Wide 

6 Frustration Threshold 1 18, as discussed above. 

7 In step 216, as part of the dialog record 106, the call handling system 104 

8 requests a response from the user 102. In step 218, the user 102 transmits the 

9 response to the call handling system 104. In step 220, if the call handling system 104 

10 can not interpret the user response, the dialog manager 108 increments the Item Wide 

11 Frustration Index 116, and compares the Item Wide Frustration Index 116 with the 

12 predetermined Item Wide Frustration Threshold 118. In step 222, if the Item Wide 

13 Frustration Index 1 16 is below the predetermined Item Wide Frustration Threshold 

14 118, the dialog manager 108 instructs the call handling system 104 to repeat the 

15 request for a user response, and the method 200 returns to step 216. 

16 In step 224, if the Item Wide Frustration Index 1 16 is at or above the 

17 predetermined Item Wide Frustration Threshold 118, the dialog manager 108: first, 

18 sends a human operator 120 the current state information from the dialog state 112; 

19 second sends the operator 120 the user responses, such as the user speech signals, 

20 stored in the dialog data cache 1 14; and third resets the Item Wide Frustration Index 

21 1 16 back to it's initial value. The current state information provides the operator 120 

22 with contextual information describing the current state of the dialog between the user 

23 102 and the call handling system 104. In step 226, the operator 120 examines the 

24 current state information and the user responses. 

25 In step 228, if the operator 120 can interpret the user responses, the operator 

26 120 provides the call handling system 104 with an interpreted response. In step 230, 

12 
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1 the call handling system 104 asks the user 102 if the interpreted response corresponds 

2 to the user's intended response (i.e. Does the user confirm that the operator's 

3 interpreted response is correct?). In step 232, if the interpreted response corresponds 

4 to the user's intended response, the call handling system 104 continues the dialog with 

5 the user 102. 

6 In step 234, if the interpreted response does not correspond to the user's 

7 intended response, the dialog manager 108 automatically commands the dialog router 

8 1 10 to interrupt the dialog between the user 102 and the system 104, and connects the 

9 user 102 directly to the operator 120. In step 236, the operator 120 enters in to a 

10 normal human dialog with the user 102 in order to determine what the user's 102 

11 intended response was. In step 238, based on the operator's 120 judgment, the 

12 operator 120 optionally connects the user 102 back to the call handling system 104 in 

13 order to complete the user's transaction. 

14 In step 240, the dialog manager 108 also automatically commands the dialog 

15 router 1 10 to connect the user 102 directly to the operator 120, if the user 102 begins 

16 to speak out of context. 

17 In step 242, the call handling system 104 creates a user specific instance of a 

18 Transaction Wide Frustration Index 122, and a set of Transaction Wide Frustration 

19 Thresholds 124, within the dialog record 106. In step 244, the dialog manager 108 

20 resets the Transaction Wide Frustration Index 122 to an initial value, such as zero, 

21 upon first contact with the call handling system 104 by the user 102. 

22 The Transaction Wide Frustration Thresholds 124 can be selected in many 

23 different ways, and may differ from user to user in a manner equivalent to that 

24 presented with respect to step 214. In step 246, if the Item Wide Frustration threshold 

25 118 has been reached or exceeded, the dialog manager 108 increments the Transaction 

26 Wide Frustration Index 122, and compares the Transaction Wide Frustration Index 

13 
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122 with a first Transaction Wide Frustration Threshold within the set of thresholds 
124. 

In step 248, if the Transaction Wide Frustration Index 122 is below the first 
Transaction Wide Frustration Threshold, the dialog manager 108 does not interrupt 
the call handling system's 104 dialog with the user 102. In step 250, if the 
Transaction Wide Frustration Index 122 is at or above the first Transaction Wide 
Frustration Threshold, the dialog manager 108 transmits a warning signal to the 
operator 120 and provides the operator 120 with an option to interrupt the dialog 
between the user 102 and the system 104, and directly connect to the user 102 through 
the dialog router 1 10. 

In step 252, if the Transaction Wide Frustration Index 122 is at or above a 
second Transaction Wide Frustration Threshold, which is higher than the first 
Transaction Wide Frustration Threshold, the dialog manager 108 automatically 
commands the dialog router 110 to connect the user 102 directly to the operator 120. 
In step 254, the operator 120 then enters in to a normal human dialog with the user 
102. 

While one or more embodiments of the present invention have been described, 
those skilled in the art will recognize that various modifications may be made. 
Variations upon and modifications to these embodiments are provided by the present 
invention, which is limited only by the following claims. 
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